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A  NETWORKED  WIZARD-OF-OZ1  EXPERIMENT  TO  STUDY  THE  EFFECT  OF 
COMMUNICATION  ON  DECISION-MAKING  PERFORMANCE 


INTRODUCTION 

The  study  of  decision  making  in  the  Combat  Information  Center  (CIC),  particularly  under  stress, 
has  been  of  particular  interest  to  the  Navy  since  the  USS  Vincennes 2  incident  [1].  Verbal 
communication  is  a  crucial  aspect  of  the  Command  and  Control  environment,  and  the 
communication  load  can  frequently  be  so  heavy  as  to  overwhelm  the  entire  CIC  process.  The  Combat 
Information  Center  [2]  of  Navy  ships  is  a  complex,  distributed  system  composed  of  multiple 
computers,  with  significant  human-computer  interaction  occurring,  and  several  voice  communication 
networks,  with  a  very  high  degree  of  human-human  communication.  This  complexity  is  compounded 
by  the  fact  that  the  major  command  communication  network  is  shared  by  multiple  personnel,  each 
competing  for  access.  Previous  studies  [3-8]  have  been  conducted  to  determine  the  type  of  work 
being  accomplished  via  verbal  communication,  the  major  participants  in  each  type  of  work,  and 
which  verbal  communications  might  be  shifted  to  the  computer  to  reduce  the  communication  load.  In 
addition  to  being  informative  for  distributing  communication  workload  in  large  teams,  the  study  of 
shifting  communication  load  from  human  to  computer  is  particularly  timely  for  reduced  manning 
efforts,  where  computers  will  be  taking  on  many  roles  previously  assigned  to  humans. 

This  software  simulation  was  developed  for  use  in  research  on  decision-making  performance 
when  acknowledgments  and  simple  replies  are  given  via  the  computer  interface  rather  than  verbally. 
This  simulation  is  based  on  a  similar  task  developed  by  Hollenbeck  and  Ilgen  [9-11]  that  was  used  by 
multiple  participants  and  aimed  primarily  at  determining  the  effect  of  distributed  decision-making. 
The  software  described  here  involves  only  one  decision  maker  who  obtains  the  information  relevant 
to  making  a  decision  from  distributed  sources. 

The  Experiment 

The  simulation  consists  of  two  networked  processes:  the  subject3  and  the  wizard.  The  participant 
plays  the  role  of  the  Anti-Air  Warfare  Officer  (AIR)  in  the  CIC.  As  such,  he4  is  given  the  task  of  a 
decision  maker  who  must  collect  information  about  two  targets  and  decide  the  threat  level  of  each 
using  a  two-tiered  decision-making  process.  The  role  of  the  two  officers  (ID  and  TRK)  from  whom 
the  information  is  collected  is  played  by  a  human-controlled  process  (the  wizard)  on  a  second 
networked  computer. 


1  Wizard-of-Oz  experiments  substitute  a  human  operator  for  some  component  of  a  computer  system  in  such  a  way 
that  experiment  participants  are  unaware  of  the  human  operator. 

2  On  July  3,  1988,  over  the  Persian  Gulf,  Iran  Air  Flight  655  was  mistakenly  identified  by  the  USS  Vincennes  as  a 
hostile  Iranian  F-14  fighter  plane  and  was  shot  down,  killing  all  290  civilians  aboard. 

3  In  the  design  of  this  experiment,  the  program,  machine,  and  processes  associated  with  the  volunteer  were  labeled 
“subject”  (those  associated  with  the  experimenter  were  labeled  “wizard”).  For  consistency,  we  continue  this  usage  in 
the  report.  When  speaking  of  the  human  volunteer,  we  use  the  term  “participant.” 

4  For  simplicity,  we  use  the  masculine  form  of  pronouns  to  refer  to  participants  (most  of  whom  were  male)  and  the 
feminine  form  for  the  wizard  (who  was  female). 
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Participants 

The  study  was  conducted  using  60  officer  volunteers,  50  males  and  10  females,  who  were 
stationed  at  the  United  States  Naval  Academy  during  the  summer  of  1997.  The  officers’  ranks 
ranged  from  newly  commissioned  01  officers  to  06s,  and  their  warfare  specialties  included  surface, 
aviation,  submarine,  Marine  Corps,  and  supply.  The  officers  were  divided  into  three  groups  based  on 
perceived  experience:  01  officers  with  no  Fleet  experience,  officers  with  surface  experience,  and  all 
other  officers.  Twenty  01  officers,  16  surface  warfare  officers,  and  24  other  officers  participated. 
Participants  were  randomly  assigned  to  each  of  the  four  modes.  Each  mode  had  15  participants. 

TASK  DESCRIPTION 

Subject  Program  Description 

On  each  trial,  the  participant  is  presented  with  two  targets,  for  which  he  must  gather  information 
on  eight  different  attributes,  determine  how  threatening  the  target  is,  and  make  a  decision  about  how 
to  deal  with  the  target.  In  making  these  decisions,  the  participant  is  instructed  to  deal  with  the  most 
threatening  target  first.  Appendix  A  contains  the  written  instructions  given  to  each  participant. 

The  simulation  has  four  possible  interaction  modes  for  collecting  the  attribute  information.  The 
responses  by  the  two  officers  controlled  by  the  wizard  can  be  presented  either  verbally  or  as  text  on 
the  computer  screen,  and  acknowledgments  can  be  made  either  verbally  or  by  lighting  a  button  on 
the  participant’s  screen.  Table  1  summarizes  the  four  combinations  of  these  modes:  Verbal 
Acknowledgment-Verbal  Reply  (VAVR),  Computer  Acknowledgment-Verbal  Reply  (CAVR), 
Verbal  Acknowledgment-Computer  Reply  (VACR),  and  Computer  Acknowledgment-Computer 
Reply  (CACR).  In  the  computer  response  conditions,  the  participant  indicates  acknowledgments  to 
the  wizard  by  selecting  the  acknowledge  button  on  the  screen;  in  the  verbal  response  conditions,  the 
participant’s  acknowledgments  are  spoken.  In  all  conditions,  the  participant  queries  the  wizard  for  the 
value  of  a  specific  attribute  by  selecting  a  labeled  button  on  the  screen  that  corresponds  to  the 
attribute  value  desired.  The  participant  requests  values  for  eight  attributes  for  each  of  the  two  targets. 
The  value  of  a  ninth  attribute,  Identification:  Friend  or  Foe  (IFF),  is  given  automatically  and  is 
presented  visually  in  all  cases. 


Table  1  —  Summary  of  the  Four  Communication  Conditions  Used  in  the  Experiment 


CACR 

CAVR 

VACR 

VAVR 

Wizard 

Acknowledge 

Button  light 

Button  light 

Verbal 

Verbal 

User  Acknowledge 

Button 

Button 

Verbal 

Verbal 

Wizard  Response 

Text 

Verbal 

Text 

Verbal 

User  Request 

Button 

Button 

Button 

Button 

The  wizard  selects  the  mode  before  the  experiment  starts.  Each  mode  generates  a  different  screen 
on  the  participant’s  machine,  as  shown  in  Figs.  1  and  2.  Labeled  buttons  are  used  to  query  the  values 
for  each  attribute.  Next  to  each  button  is  a  text  box  that  serves  two  purposes:  the  participant  may  use 
the  box  to  enter  information  that  is  received  verbally,  or,  in  the  Computer  Reply  conditions,  the  value 
of  the  requested  attribute  appears  in  the  corresponding  text  box.  The  participant  may  also  enter  the 
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Fig.  1  —  The  subject’s  screen  for  verbal  acknowledgment  modes  VACR  and  VAVR 


Fig.  2  —  The  subject’s  screen  for  computer  acknowledgment  modes  CACR  and  CAVR 
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threat  level  of  an  attribute  by  clicking  on  one  of  three  threat-level  boxes:  very  threatening,  somewhat 
threatening,  or  nonthreatening.  A  clock  shows  the  time  elapsed  since  the  start  of  the  trial  and  is 
included  to  provide  time  pressure,  even  though  there  are  no  actual  consequences  for  not  responding 
quickly.  The  screen  also  provides  a  decision  box  for  each  target  and  the  information  needed  to 
evaluate  threat  levels  and  reach  a  decision. 

The  wizard  controls  information  from  both  the  ID  and  the  TRK  officers,  but  it  appears  to  the 
participant  as  if  he  is  requesting  information  from  two  different  team  members.  In  the  Verbal 
Response  versions,  the  two  officers  are  represented  by  two  different  voices.  Requests  for  information 
about  Altitude,  Angle,  Speed,  and  Size  are  made  to  the  ID  officer  and  requests  for  information  about 
Corridor,  Range,  Bearing,  and  Radar  Class  are  made  to  the  TRK  officer.  Information  about  the  ninth 
attribute,  IFF,  is  always  provided  visually  on  the  screen  and  does  not  have  to  be  requested.  When  the 
simulated  officers  see  the  request  for  information,  they  immediately  acknowledge  the  request.  For 
added  realism,  delays  are  sometimes  included  to  simulate  the  fact  that  these  officers  often  have  other 
demands  on  their  time.  For  example,  they  may  need  to  obtain  the  information  from  other  sources  or 
may  be  responding  to  other  requests  in  the  CIC.  The  reply  to  a  request  may  be  immediate,  have  a 
short  delay  (3.33  s),  or  have  a  long  delay  (6.67  s).  On  each  trial,  two  randomly  selected  attributes  for 
each  officer  will  have  a  short  delay,  two  will  have  a  long  delay,  and  the  remaining  four  attributes  will 
be  given  immediate  responses.  Also,  to  mimic  the  effects  of  human  fallibility,  on  two  of  the  eight 
requests,  the  simulated  TRK  does  not  “notice”  the  request  and  fails  to  acknowledge  it.  This  simulates 
the  effects  of  high  workload  or  inexperience  and  requires  the  participant  to  repeat  the  request  in 
order  to  get  the  information.  TRK  always  sees  the  second  request  and  acknowledges  it  immediately. 

The  participant  must  acknowledge  the  receipt  of  information  in  reply  to  each  request.  If  he  fails 
to  do  so  within  8.33  s,  a  dialog  box  appears,  asking  if  he  received  the  reply  for  the  specific  attribute. 
When  the  dialog  box  is  on  the  screen,  no  other  work  can  be  done.  If  the  participant  answers  yes,  he 
may  continue  to  work.  If  he  answers  no,  the  ID  or  TRK  officer  will  send  the  information  again. 

Text  boxes  are  provided  for  each  attribute.  In  the  Computer  Reply  modes,  the  reply  for  each 
attribute  automatically  appears  in  the  corresponding  text  box.  In  the  Verbal  Reply  modes,  the 
participant  has  the  option  of  typing  the  reply  in  the  appropriate  text  box.  For  each  attribute,  there  are 
also  check  boxes  labeled  V,  S,  and  N  (for  very  threatening,  somewhat  threatening,  and 
nonthreatening)  that  can  be  used  to  record  the  threat  level  of  the  attribute.  The  participant  may 
change  the  selection  after  a  box  has  been  selected  or  choose  not  to  use  the  boxes  at  all. 

There  are  five  possible  decisions  for  a  target,  in  order  of  increasing  severity:  Track,  Watch, 
Report,  Assign,  and  Engage.  The  decision  for  each  target  reflects  the  severity  of  the  threat  it  poses, 
based  on  a  two-tiered  decision  process  that  combines  the  threat  levels  of  the  nine  attributes  for  that 
target.  A  threat  score  is  calculated  by  first  determining  the  threat  level  (very  threatening,  somewhat 
threatening,  or  nonthreatening)  for  each  of  nine  attributes  and  then  combining  pairs  of  attributes 
according  to  a  simple  formula. 

The  threat  levels  for  the  individual  attributes  (Altitude,  Angle,  Speed,  Size,  Corridor,  Range, 
Bearing,  Radar  Class,  and  IFF)  are  determined  by  comparing  their  values  with  the  table  of 
“standard”  classifications  (Table  2)that  is  provided  for  reference  at  the  bottom  of  the  participant’s 
screen.  Each  threat  level  is  associated  with  a  cue  value,  as  shown  in  the  bottom  line  of  Table  2.  A  very 
threatening  attribute  is  assigned  a  cue  value  of  two;  a  somewhat  threatening  attribute,  a  value  of  one; 
and  a  nonthreatening  attribute,  a  value  of  zero.  These  cue  values  are  combined,  using  the  formula 
below,  to  determine  the  overall  threat  posed  by  the  target. 

Threat  Level  =  (2  *  IFF  Cue)  +  (Altitude  Cue  *  Corridor  Cue)  +  (Angle  Cue  *  Range 
Cue)  +  (Speed  Cue  *  Bearing  Cue)  +  (Size  Cue  *  Radar  Cue) 
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Table  2  —  Standard  Classification  Chart 


Degree  of  Threat 

Attributes 

Very  Threatening 

Somewhat  Threatening 

Nonthreatening 

5,000  to  13,000  ft 

17,000  to  23,000  ft 

27,000  to  35,000  ft 

Angle 

o 

OO 

i 

2 

o 

in 

i 

-3°  to  +3° 

+8°  to  +15° 

Speed 

600  to  800  mph 

500  to  325  mph 

275  to  100  mph 

Size 

10  to  17  m 

23  to  37  m 

43  to  65  m 

IFF 

1.4  to  1.8  MHz 

1.1  to  0.9  MHz 

0.6  to  0.2  MHz 

Corridor 

30  to  22  mi 

18  to  12  mi 

8  to  0  mi 

Range 

1  to  40  mi 

60  to  90  mi 

110  to  200  mi 

Direction 

o 

o 

o 

O 

O 

OO 

o 

12°  to  18° 

22°  to  30° 

Radar 

Class  9  and  8 

Class  5 

Class  2  and  1 

Cue 

2 

1 

0 

Unlike  in  real-world  situations,  each  pair  of  attributes  is  weighted  equally.  This  simplification  was 
made  so  that  novices  not  experienced  with  Navy  command  and  control  decision  making  could  use 
the  task.  Even  though  the  decision  formula  has  been  simplified  to  accommodate  users  without  Navy 
experience,  the  decision  task  retains  enough  realism  to  be  acceptable  to  more  experienced  users.  The 
formula  arranges  the  attributes  in  pairs  because  they  are  related.  For  example,  the  relationship 
between  Angle  and  Range  can  be  seen  in  Table  3.  Individually,  Angle  or  Range  values  may  appear  to 
be  threatening,  but  the  combination  may  or  may  not  be  threatening,  depending  on  the  value  of  the 
associated  attribute. 


Table  3  —  The  Relationship  between  Angle  and  Range 


Range 

Angle 

Descending 

Level 

Ascending 

Close 

Very  threatening 

Moderately  threatening 

Nonthreatening 

Intermediate 

Moderately  threatening 

Somewhat  threatening 

Nonthreatening 

Distant 

Nonthreatening 

Nonthreatening 

Nonthreatening 

Similar  relationships  exist  between  Altitude  and  Corridor,  Speed  and  Bearing,  and  Size  and  Radar 
Class.  Table  4  presents  examples  of  threatening  and  nonthreatening  combinations  of  attribute  values. 
Multiplying  the  cue  value  of  one  attribute  by  the  cue  value  of  its  paired  attribute  yields  a  number 
between  0  (nonthreatening)  and  4  (very  threatening)  that  indicates  the  impact  of  the  two  attributes  on 
the  decision.  Note  that  if  either  attribute  has  a  zero  cue  value,  the  combination  is  nonthreatening. 
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Table  4  —  Additional  Examples  of  How  the  Combined  Values  of  Attributes  Determine  Threat  Levels 


Attribute 

Very  Threatening 

Nonthreatening 

Nonthreatening 

Altitude 

low-flying  target 

high-flying  target 

low-flying  target 

Corridor 

outside  the  corridor 

outside  the  corridor 

middle  of  corridor 

Speed 

fast  target 

fast  target 

slow  target 

Bearing 

headed  toward  ship 

headed  away  from  ship 

headed  toward  ship 

Size 

small  target 

small  target 

large  target 

Radar  Class 

weapons  radar 

weather  radar 

weapons  radar 

To  determine  the  impact  of  the  IFF  value  on  the  decision,  the  cue  value  is  doubled.  If  the  IFF 
value  identifies  a  target  as  a  military  target  (very  threatening  with  cue  of  2),  it  is  especially 
threatening,  whereas  a  civilian  target  (nonthreatening  with  cue  of  0)  is  not. 

The  threat  level  formula  produces  a  number  between  0  and  20.  The  decision  about  how  to 
handle  the  target  can  then  be  determined  from  Table  5. 


Table  5  —  Values  for  Making  the  Target  Decisions 


Decision 

Threat  Level 

Track 

0-5 

Watch 

6-9 

Report 

10-14 

Assign 

15-17 

Engage 

18-20 

A  decision  to  Track  a  target  means  that  no  further  attention  needs  to  be  given  to  this  target  but 
the  target’s  ID  should  remain  visible.  A  decision  to  Watch  a  target  implies  that  the  target  should 
continue  to  be  watched  on  radar.  A  decision  to  Report  a  target  means  the  ship  should  send  a  message 
to  the  target  identifying  the  ship  and  alerting  the  target  to  steer  clear.  A  decision  to  Assign  the  target 
implies  the  ship  should  synchronize  its  radar  and  attack  weapons  so  that  the  weapons  are  fixed  on  the 
target.  A  decision  to  Engage  a  target  requires  weapons  to  be  launched. 

The  participant  makes  a  decision  concerning  the  threat  level  of  the  target  by  selecting  a  radio 
button.  Although  participants  are  told  to  deal  with  the  most  threatening  target  first,  the  order  of  the 
decisions  has  no  visible  consequences  and  the  participant  may  change  his  mind  on  the  first  decision. 
However,  as  soon  as  the  second  decision  is  made,  the  trial  ends. 


A  Networked  Wizard-of-Oz  Experiment 


1 


Wizard  Program  Description 

Before  each  participant  begins,  the  wizard  selects  one  of  the  four  modes  from  the  Mode  Menu. 
The  wizard  also  selects  the  number  of  trials  from  a  menu  when  the  participant  begins.  If  Practice  is 
selected,  the  participant  is  given  two  trials.  If  Real  Trial  is  selected,  then  the  participant  is  given  1 0 
trials.  Appendix  B  lists  the  attribute  values  used  in  the  practice  (one  scenario,  repeated)  and  the 
experiment  trials. 

Although  it  appears  to  the  participant  that  he  is  receiving  attribute  information  from  two  separate 
officers,  the  wizard  is  actually  functioning  as  both  officers.  The  wizard’s  screen  (Fig.  3)  is  divided 
into  quarters.  Four  attributes  for  the  first  target  are  in  the  upper  left  quadrant,  and  the  same  four 
attributes  for  the  second  target  are  in  the  upper  right  quadrant.  The  ID  officer  controls  these  eight 
attributes.  The  other  four  attributes  for  the  first  target  are  in  the  lower  left  quadrant,  and  the  other 
four  attributes  for  the  second  target  are  in  the  lower  right  quadrant.  These  are  the  attributes  controlled 
by  the  TRK  officer.  The  ninth  attribute,  IFF,  does  not  appear  because  it  is  displayed  automatically  on 
the  participant’s  screen  and  does  not  require  the  wizard’s  intervention. 


•  -  -.tr-fc  i»nuR _ ; 

Target  1  ID  Target  2 


Altitude  Angle  Speed  Size 

Altitude  Angle  Speed  Size 

[  Reply  ]  [  Reply  ]  (  Reply  ]  [  Reply  ] 

(  Reply  ]  |  Reply  )  (  Reply  ]  [  Reply  ] 

(  Repeal  ]  [  Repeat  ]  [  Repeal  ]  [  Repeat  ] 

(  Repeal  ]  [  Repeat  ]  [  Repeal  J  [  Repeat  ] 

[flcknRec]  [flcknRec]  [ficknRec]  [flcknRec] 

[flcknRec]  [fickn  Rec  |  [flcknRec]  [RcknRec] 

[  alert  }  [  flleil  )  (  alert  ]  (  flleil  ) 

(  Rlert  ]  (  flleil  ]  (  filer!  ]  [  flleil  ) 

Target  1  target  2 


Corridor 

Range 

Bearing 

Radar 

Corridor 

Range 

Bearing 

Radar 

i— ■■ 

IHMI 

(  Reply  ) 

(  Reply  ] 

[  Reply  ] 

(  Reply  ) 

[  Reply  1 

nun 

[  Repeat  ] 

[  Repeat  ] 

IH3SB1 

■missal 

[fickn  Rec] 

[fickn  Rec ] 

■032103 1 

[fickn  Rec  ] 

[fickn  Rec ] 

■0333331] 

(  flleil  ] 

[  flleil  ] 

IHMl 

■■nay 

[  flleil  ] 

Fig.  3  —  The  wizard’s  screen 


Each  attribute  has  four  buttons  associated  with  it.  The  Reply  button  becomes  highlighted  when 
the  participant  requests  information;  the  wizard  then  selects  (clicks)  the  button  and  this  causes  the 
program  to  send  an  encoded  message  to  the  subject  machine.  When  the  subject  machine  receives  this 
message,  the  program  immediately  activates  the  acknowledgment  of  the  participant’s  request  and 
determines  whether  to  reply  immediately,  after  a  short  delay  (3.33  s),  or  after  a  long  delay  (6.67  s).  If 
the  reply  should  be  delayed,  then  an  alarm  is  set  on  the  subject  machine.  Each  time  the  Event  Loop 
enters  an  idle  cycle,  the  delayed  replies  are  checked  to  determine  if  it  is  time  to  reply  to  the 
participant.  When  the  delay  time  has  elapsed,  the  reply  is  finally  given  to  the  participant  and  an  alert 
alarm  is  set.  If  the  participant  does  not  acknowledge  the  reply  before  the  alert  alarm  goes  off,  then  an 
encoded  message  is  sent  to  the  wizard  machine  that  highlights  that  attribute’s  Alert  button. 
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When  the  participant  acknowledges  a  reply,  an  encoded  message  is  sent  to  the  wizard  machine. 
The  wizard  machine  decodes  the  message  and  highlights  the  appropriate  attribute’s  AcknRec  button. 
When  the  wizard  selects  the  AcknRec  button,  it  is  disabled  and  an  alert  flag  for  that  attribute  is  set. 
When  the  wizard  machine  receives  an  alert  message,  it  determines  whether  the  alert  flag  has  been  set. 
If  it  has,  it  is  toggled  and  the  alert  is  ignored.  If  it  has  not  been  set,  then  the  Alert  button  is 
highlighted. 

When  the  wizard  selects  the  Alert  button,  an  encoded  message  is  sent  to  the  subject  machine  that 
immediately  displays  a  dialog  box.  The  dialog  box  asks  the  participant  if  he  received  the  response  for 
the  appropriate  attribute.  The  participant  must  respond  either  yes  or  no  to  the  dialog  box  before  he 
can  continue.  His  answer  is  then  encoded  and  sent  to  the  wizard  machine.  If  the  participant  responds 
yes,  then  nothing  else  occurs.  If  he  responds  no,  then  the  Repeat  button  for  the  appropriate  attribute 
is  highlighted  on  the  wizard’s  screen.  The  wizard  selects  the  Repeat  button  and  an  encoded  message 
is  sent  to  the  subject  machine.  The  subject  machine  decodes  the  message  and  again  determines 
whether  the  reply  should  be  immediately  delivered  or  delayed.  If  it  is  delayed,  then  the  delay  routine 
is  executed.  If  the  participant  fails  to  acknowledge  this  reply,  he  will  receive  another  alert  dialog  box. 

When  the  participant  has  completed  both  decisions,  an  encoded  message  is  sent  to  the  wizard 
machine.  This  machine  then  either  starts  another  trial  or  sends  a  message  to  the  subject  machine  to 
display  a  summary  of  the  correct  decisions  and  the  participant’s  decisions  for  each  trial. 

Modes 

As  each  new  participant  begins,  the  wizard  must  select  a  mode  type  from  the  Mode  Menu.  Four 
different  modes  are  available:  Verbal  Acknowledgment-Verbal  Reply  (VAVR);  Computer 
Acknowledgment-Verbal  Reply  (CAVR);  Verbal  Acknowledgment-Computer  Reply  (VACR);  and 
Computer  Acknowledgment-Computer  Reply  (CACR).  Each  mode  will  bring  up  a  different  screen 
on  the  subject  machine  (Figs.  1  and  2). 

All  of  the  wizard’s  verbal  acknowledgments  and  verbal  replies  are  prerecorded  sound  files  that 
are  stored  on  the  subject  machine.  Whenever  the  wizard  selects  a  Reply  button,  an  encoded  message  is 
sent  to  the  subject  machine.  When  the  subject  machine  decodes  this  message,  it  immediately 
acknowledges  the  participant.  In  the  two  Computer  Acknowledgment  modes,  CAVR  and  CACR,  the 
RequestAckned  button  for  the  appropriate  officer  is  highlighted  on  the  participant’s  screen  for  a 
short  period  of  time.  In  the  two  Verbal  Acknowledgment  modes,  VAVR  and  VACR,  the  sound 
resource  for  the  appropriate  officer  is  played.  There  is  no  RequestAckned  button  on  screens  for  the 
VAVR  and  VACR  modes. 

In  addition  to  determining  what  delay  to  use  before  the  reply  is  given,  the  program  on  the 
subject  machine  also  selects  the  mode  for  the  reply.  For  VAVR  and  CAVR,  the  appropriate  sound 
resource  is  played;  for  VACR  and  CACR,  the  reply  is  placed  in  the  text  box  beside  the  name  of  the 
attribute.  There  is  a  text  box  on  the  participant’s  screen  beside  each  attribute  name,  whether  the  reply 
is  given  verbally  or  as  text,  and  in  either  case  it  is  possible  for  the  participant  to  enter  information  in 
the  box  if  desired.  It  is  not  necessary  for  the  participant  to  enter  this  information;  the  box  is  simply 
provided  as  a  memory  aid. 

The  participant  must  acknowledge  each  reply  after  it  is  received.  In  the  CAVR  and  CACR  modes, 
once  the  reply  is  delivered,  a  SendAckn  button  corresponding  to  the  attribute  is  highlighted.  This 
button  remains  highlighted  until  the  participant  selects  it  or,  if  the  participant  does  not  acknowledge 
within  the  required  time,  until  an  alert  alarm  is  triggered.  While  the  button  is  enabled,  the  participant 
can  select  it,  thereby  sending  an  encoded  message  to  the  wizard’s  machine  that  her  reply  has  been 
acknowledged.  In  the  VAVR  and  VACR  modes,  the  participant  must  verbally  acknowledge  the  reply; 
he  does  so  by  speaking  the  name  of  the  attribute  and  the  word  “aye.”  For  altitude,  the  participant 
would  respond  with  “Altitude  Aye.”  In  the  Verbal  Reply  modes,  the  AcknRec  buttons  on  the  wizard 
machine  are  enabled  when  the  trial  begins.  After  hearing  the  participant  verbally  acknowledge  the 
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reply,  the  wizard  must  select  the  AcknRec  button.  If  the  AcknRec  button  is  not  selected,  an  alert  alarm 
is  triggered.  As  described  earlier,  the  alert  alarm  activates  a  dialog  box  on  the  participant’s  screen  that 
asks  if  the  reply  was  received. 

The  participant  wears  earphones  for  all  four  modes.  This  is  realistic  for  the  CIC  environment  and 
also  helps  to  filter  out  distracting  noises  when  the  experiment  is  not  conducted  in  soundproof  rooms. 
Sound  files  for  the  VAVR,  VACR,  or  CAVR  modes,  and  also  the  alert  signal  for  the  dialog  box  in  all 
modes,  are  heard  through  the  earphones.  An  added  benefit  to  using  earphones  to  deliver  the  sound 
files  is  that  the  wizard  does  not  hear  them  and  thus  is  better  able  to  concentrate  and,  where 
appropriate,  to  listen  for  verbal  acknowledgments. 

Data  Collection 

Both  the  subject  and  the  wizard  machine  maintain  time-stamped  log  files  of  everything  the 
participant  and  the  wizard  do  during  the  experiment.  Both  systems  also  produce  summary  files.  The 
subject  summary  file  contains  the  mode,  the  trial  number,  the  attributes  with  a  short  delay  before  a 
reply  is  delivered,  the  attributes  with  a  long  delay  before  a  reply  is  delivered,  the  correct  threat-level 
decision,  the  participant’s  threat-level  decisions,  and  the  elapsed  time  in  tick  counts.  The  summary 
file  records  the  number  of  times  each  attribute  was  requested,  the  number  of  acknowledgments  sent, 
the  number  of  attribute  threat-level  check  boxes  used,  the  number  of  times  the  participant  changed 
his  mind  about  the  attribute  threat-level,  and  the  number  of  text  boxes  used.  The  wizard  summary  file 
contains  the  mode,  the  trial  number,  and  the  two  TRK  attributes  that  needed  a  second  request.  In 
addition,  for  each  attribute  it  records  the  number  of  attribute  requests,  the  number  of  replies,  the 
number  of  visual  and  verbal  acknowledgments  received,  the  number  of  alerts,  the  number  of  replies 
the  participant  received  but  failed  to  acknowledge,  and  the  number  of  replies  the  participant  failed  to 
acknowledge  because  he  did  not  see/hear  them.  Appendix  C  presents  a  sample  of  the  data  files. 

SOFTWARE  AND  HARDWARE  CONFIGURATION 

Software  Development 

This  software  was  developed  in  C++  using  the  Metro  werks  Code  Warrior  11  Integrated 
Development  Environment  on  a  Power  Macintosh  8500/180  with  98  MB  RAM  using  68K  emulation 
and  a  Macintosh  Quadra  800/33  with  48  MB  RAM.  The  code  files  are  available  on-line  at 
<http://www.aic.nrl.navy.mil/papers/1999/>.  The  communication  software  used  is  a  modification  of  the 
CTB  code  developed  by  Mark  Chally  for  his  Happy  Face  program  [12]  and  can  be  downloaded  from 
<http://home.earthlink.net/~chally/>.  The  asynchronous  sound  software  used  is  ASndPlayer.c,  which 
was  developed  by  Manuel  Perez-Quinones  as  part  of  the  Intelligent  Cockpit  Project  at  the  Naval 
Research  Laboratory  [13]. 

The  Wizard  application  needs  347  KB  of  storage  while  the  Subject  application  requires  7.5  MB 
of  storage,  mainly  due  to  the  large  size  of  the  sound  resource  file.  If  it  is  necessary  to  change  the 
source  code  for  thus  simulation,  the  following  things  should  be  noted.  The  CodeWarrior  IDE  partition 
size  was  set  at  7  MB  for  the  Wizard  code  and  26  MB  for  the  Subject  code.  If  the  source  code  needs  to 
be  edited  and  the  debugger  used,  the  size  of  the  partitions  needs  to  be  increased.  Virtual  memory  was 
turned  on  for  both  machines  and  the  RAM  allocation  thereby  doubled.  The  minimum  heap  size, 
found  in  the  Edit/Project  Settings/Project/68K  Project  menu,  was  set  at  3  MB  for  the  Wizard  and  21 
MB  for  the  Subject.  The  heap  in  the  Subject  code  is  that  large  due  to  the  sound  resources  embedded 
in  the  Subject  project.  If  the  source  code  is  recompiled  and  a  link  error  occurs  that  indicates 
insufficient  space  in  the  heap  zone,  the  heap  size  must  be  increased. 

If  modifications  are  made  to  the  communications,  the  CTB  code  may  need  to  be  modified.  The 
CTB  code  only  allows  for  one  byte  to  be  passed  between  machines.  Since  there  are  eight  attributes, 
numbered  0  to  7,  and  two  targets,  numbered  0  to  1,  it  takes  three  bits  to  pass  the  attribute  number  and 
one  bit  to  pass  the  target  number.  This  only  leaves  four  bits  to  encode  the  type  of  message  being 
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passed,  thereby  limiting  the  number  of  different  types  of  messages  that  can  be  passed  between  the  two 
machines.  This  limitation  had  a  definite  impact  on  the  design  of  the  simulation. 

Appendix  D  describes  the  networking  configuration  and  initialization  procedures  for  execution 
of  the  program. 

Hardware  Configuration 

The  computers  were  networked  together  with  a  LocalTalk  cable  connecting  their  printer  ports.  If 
the  machines  are  already  connected  to  Ethernet,  that  connection  may  remain  in  place.  The  Mac-to- 
Mac  LocalTalk  cable  may  also  remain  in  place.  Four  extensions  were  added  to  the  Extensions  Folder 
of  the  System  Folder.  ADSP  and  AppleTalk  ADSP  Tool  are  necessary  for  the  communications  to 
work  and  are  available  on  the  Macintosh  homepage.  MetroNub  is  required  if  it  necessary  to  recompile 
or  edit  the  source  code;  CFM-68K  Runtime  Enabler  is  also  required  if  the  Macintosh  is  not  a 
PowerPC  model.  These  are  provided  in  Metrowerks  CodeWarrior  11. 

RESULTS 

Several  performance  measures  were  collected  and  analyzed.  These  included  the  total  time  to 
complete  each  scenario,  the  number  of  correct  decisions,  and  the  number  of  repeated  requests.  A 
more  detailed  analysis  of  the  order  in  which  attributes  were  requested  was  used  to  assess  user 
strategies  in  collecting  target  information. 

Time  to  Complete  the  Task 

The  time  to  complete  the  task  was  faster  for  text  (computer)  responses  than  for  verbal  responses 
(Fig.  4),  and  there  was  an  overall  trend  toward  completing  the  task  faster  over  trials.  Whether 
acknowledgments  were  made  verbally  or  by  computer  had  little  effect  on  the  time  it  took  to  complete 
the  task.  However,  for  the  last  trials,  when  participants  were  the  most  practiced,  the  VAVR  group  was 
the  slowest  to  complete  the  task.  An  analysis  of  variance  (Table  6)  showed  significant  effects  due  to 
response  mode,  F  (1,  56)  =  27.45,  p  <  0.001,  and  trials,  F  (9.  504)  =  46.58,  p  <  0.001,  and  no  other 
significant  effects  or  interactions. 

Participants  may  have  taken  longer  in  the  verbal  response  conditions  due  in  part  to  the  fact  that 
responses  took  longer  to  say  than  to  display,  and  this  would  also  be  true  when  verbal  responses  are 
used  in  real  situations.  However,  a  greater  proportion  of  the  additional  task  time  was  probably  taken 
up  by  the  participants  typing  in  the  spoken  attribute  values.  The  task  could  be  performed  accurately 
without  this  extra  typing  by  simply  clicking  the  appropriate  threat  level  box  corresponding  to  the 
attribute  value,  given  that  threat  levels,  not  attribute  values,  were  used  in  the  formula  used  to  decide 
how  to  deal  with  the  target.  However,  most  people  in  the  verbal  response  conditions  typed  in  the 
attribute  values  and  only  looked  up  the  threat  values  after  they  had  collected  the  attribute  data.  The 
verbal  response  condition  caused  the  user  to  take  extra  steps  that  were  not  needed  in  the  computer 
response  condition. 
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Trial  number 


Fig.  4  —  Average  time  to  complete  the  task  over  trials  for  each  group 


Table  6  —  Analysis  of  Variance  for  Total  Elapsed  Time 


BETWEEN  SUBJECTS 

SOURCE 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F  Ratio 

Probability 

Ackn.  format 

7453107 

1 

7453107 

0.276 

0.602 

Resp.  format 

741371000 

1 

741371000 

27.425 

0.000 

Ackn.  format 

X  Resp.  format 

12210000 

1 

12210000 

0.452 

0.504 

Error 

1513820000 

56 

27032600 

WITHIN  SUBJECTS 

SOURCE 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F  Ratio 

Probability 

Trials 

753835000 

9 

83759500 

46.578 

0.000 

Trials  X  Ackn.  format 

4386244 

9 

487360 

0.271 

0.982 

Trials  X  Resp.  format 

20372200 

9 

2263580 

1.259 

0.257 

Trials  X  Ackn.  format 

X  Resp.  format 

8815314 

9 

979479 

0.545 

0.842 

Error 

906330000 

504 

1798273 
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Correct  Decisions 

Communication  mode  had  little  or  no  effect  on  the  number  of  correct  decisions  (Fig.  5  and 
Table  7),  and  the  number  of  correct  decisions  also  did  not  change  systematically  over  trials.  A 
2x2x10  analysis  of  variance  (Table  8)  showed  no  significant  effects  or  interactions  due  to 
acknowledgment  mode,  response  mode,  or  trials  (F  <  1.0  for  all  effects).  This  is  a  very  positive  result 
and  indicates  that  the  officers  were  able  to  make  good  decisions  even  under  conditions  where 
communicating  required  more  time  and  effort. 


Fig.  5  —  Average  number  of  correct  responses  by  trials  for  each  of  the  four  groups 


Table  7  —  Total  Number  of  Correct  Responses  for  Each  Group 


Acknowledge 

Computer 

Verbal 

Response 

Mean 

Std.  Error 

Mean 

Std.  Error 

Text 

18.00 

0.414 

17.13 

0.380 

Verbal 

17.80 

0.380 

17.00 

0.602 
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Table  8  —  Analysis  of  Variance  for  Number  of  Correct  Responses 


SOURCE 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F  Ratio 

Probability 

Ackn.  format 

10.417 

1 

10.417 

2.039 

0.159 

Resp.  format 

0.417 

1 

0.417 

0.082 

0.776 

Ackn.  format 

X  Resp.  format 

0.017 

1 

0.017 

0.003 

0.955 

Error 

286.133 

56 

5.110 

Repeated  Requests 

When  the  response  is  delayed,  a  request  may  be  repeated,  either  because  the  participant  thinks  the 
request  has  not  been  received  or  because  the  requested  information  may  have  been  forgotten. 
Response  format  significantly  influenced  the  number  of  repeated  requests,  with  more  repeated 
requests  for  verbal  responses.  As  expected,  the  number  of  repeated  requests  increased  significantly  as 
a  function  of  delay  and  there  was  also  a  significant  interaction  of  delay  with  acknowledgment  format 
(Fig.  6  and  Table  9). 

When  there  is  no  delay,  the  attribute  information  arrives  almost  immediately,  so  the 
acknowledgment  modality  has  little  effect.  However,  the  verbal  response  has  an  increased  probability 
of  being  forgotten,  and  for  this  reason,  the  attribute  request  may  need  to  be  repeated.  Verbal 
responses  are  more  transient,  and  the  specific  new  information  that  is  conveyed  about  the  value  of  the 
attribute  that  was  queried  is  therefore  more  likely  to  be  forgotten  (or  lost  from  short-term  memory). 
This  is  even  more  of  a  problem  if  there  is  a  delay  between  the  query  and  the  response.  There  is  then 
the  added  possibility  that  the  query  is  no  longer  actively  in  short-term  memory,  and  the  user  must 
then  register  both  which  attribute  is  being  addressed  and  its  value. 

Verbal  acknowledgments  (no  new  information)  are  more  likely  to  capture  attention,  even  when 
the  person  is  looking  at  another  part  of  the  screen  and  are,  therefore,  more  likely  to  be  noticed.  With 
short  delays,  a  verbal  acknowledgment  is  more  likely  to  have  been  noticed  than  a  computer 
acknowledgment  and  the  person  is  less  likely  to  repeat  a  request,  whereas  if  the  computer 
acknowledgment  isn’t  noticed,  they  are  more  likely  to  repeat  the  request  because  they  think  it  has  not 
been  received.  With  long  delays,  an  added  negative  effect  of  verbal  acknowledgments  is  that  the 
person  may  no  longer  remember  which  attribute  they  requested.  If  the  query  is  no  longer  actively  in 
short-term  memory,  the  person  must  then  process  which  attribute  is  being  addressed  as  well  as  its 
value.  Also  with  long  delays,  there  is  a  tradeoff  between  the  positive  effects  of  verbal 
acknowledgments  and  the  negative  effects  of  verbal  responses. 
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Delay 

Fig.  6  —  Average  number  of  requests  per  attribute  for  each  of  the  four  groups 


Table  9  —  Analysis  of  Variance  for  Number  of  Repeated  Requests 


BETWEEN  SUBJECTS 

SOURCE 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F  Ratio 

Probability 

Ackn.  format 

0.029 

1 

0.029 

1.702 

0.197 

Resp.  format 

0.097 

1 

0.097 

5.702 

0.020 

Ackn.  format 

X  Resp.  format 

0.000 

1 

0.000 

0.001 

0.972 

Error 

0.957 

56 

0.017 

WITHIN  SUBJECTS 

SOURCE 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F  Ratio 

Probability 

Delay 

0.134 

2 

0.067 

13.559 

0.000 

Delay*Ackn.  format 

0.039 

2 

0.020 

3.953 

0.022 

Delay*Resp.  format 

0.007 

2 

0.003 

0.696 

0.501 

Delay* Ackn.  format 
*Resp.  format 

0.003 

2 

0.002 

0.332 

0.718 

Error 

0.554 

112 

0.005 
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Strategies  for  Evaluating  Targets 

Participants  were  told  to  deal  with  the  most  threatening  target  first.  There  were  individual 
differences  in  collecting  information  about  the  two  threats,  as  well  as  changes  over  time  (Fig.  7).  The 
number  of  attributes  requested  before  completing  the  decision  on  the  most  threatening  target 
generally  tended  to  decrease  over  trials,  with  the  exception  of  scenarios  2  and  10.  All  four  groups 
showed  the  same  pattern  of  change  over  trials. 


■  ■  Threat  target 

-  -  O-  -  NT  target 


Fig.  7  —  Total  number  of  attribute  requests  before  decision  for  each  target  type 


Table  10  —  Number  of  Trials  on  which  All  Attributes  on  Both  Targets  Were  Requested  Before 

Making  the  Decision  on  Either  Target 


Acknowledge 

Computer 

Verbal 

Response 

Mean 

Std.  Error 

Mean 

Std.  Error 

Text 

5.00 

1.01 

3.87 

0.89 

Verbal 

3.73 

0.80 

4.60 

0.99 

Overall,  the  mode  of  communication  did  not  significantly  affect  the  order  in  which  decisions 
were  made  about  each  of  the  two  targets.  As  shown  above  in  Table  10,  participants  in  the  CAVR  and 
VACR  conditions  were  somewhat  more  likely  to  request  all  attributes  before  making  a  decision  on  the 
most  threatening  target,  and,  as  a  result,  they  also  made  fewer  threat  decisions  before  requesting  the 
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remaining  attributes  (Table  11).  However,  these  differences  were  not  consistent  across  either  response 
mode  or  acknowledge  mode,  and  they  were  relatively  small  and  were  not  statistically  significant. 


Table  11  —  Number  of  Trials  on  which  the  Decision  on  the  Most  Threatening  Targets  Was  Made 
Before  Requesting  the  Remaining  Attributes  on  the  Less  Threatening  Target 


Acknowledge 

Computer 

Verbal 

Response 

Mean 

Std.  Error 

Mean 

Std.  Error 

Text 

4.47 

0.81 

5.33 

0.75 

Verbal 

5.53 

0.68 

4.73 

0.90 

CONCLUSIONS 

This  software  was  developed  to  study  the  effects  of  communication  mode  on  information 
gathering  and  decision  making  in  a  command  and  control  context.  A  study  using  this  software  was 
conducted  using  Navy  officers  stationed  at  the  United  States  Naval  Academy.  The  results  of  the  study 
showed  no  significant  change  in  the  accuracy  of  the  threat-level  decision  made,  but  strategies  and  the 
time  it  takes  to  make  the  decision  were  significantly  affected  by  communication  mode. 

•  Communication  mode  affects  information  gathering  strategy  and  the  time  taken  to  complete 
the  task,  but  it  does  not  affect  threat  level  assessment  or  the  quality  of  the  final  decisions. 

•  Presenting  numerical  information  in  text  or  visual  form  helps  to  reduce  memory  load  and 
reduces  the  need  for  repeated  requests. 

•  In  cases  where  there  is  a  delay  before  a  response  is  received,  acknowledgments  are  very 
important  and  they  should  be  in  a  form  that  will  be  attended.  Unless  auditory  overload  is  a 
serious  problem,  verbal  acknowledgments  are  preferable. 

The  results  of  this  study  will  be  used  to  determine  the  feasibility  of  transferring  certain 
information  that  is  currently  delivered  verbally  in  the  CIC  to  the  computer  interface.  This  has 
important  implications  for  reduced  manning  efforts.  A  second  study  using  civilians  will  compare  the 
performance  of  civilian  users  with  that  of  Navy  officers.  The  outcome  of  the  second  study  will  have 
important  implications  for  the  Navy’s  use  of  civilians  in  research  to  test  the  technology  that  is  to  be 
used  in  the  Fleet. 
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Appendix  A 

PARTICIPANT  INSTRUCTIONS 


INSTRUCTIONS 
All  Conditions 

In  this  experiment  you  will  be  playing  the  role  of  an  Anti- Air  Warfare  Officer  deciding  how  to 
interact  with  two  unidentified  targets  in  your  airspace.  You  will  make  this  decision  by  combining 
information  about  nine  attributes  of  the  target.  You  will  request  this  information  from  “ID”  and 
“TRK”  officers  in  the  Combat  Information  Center  (CIC).  As  shown  on  your  screen,  the  attributes  are 
Altitude,  Angle,  Speed,  Size,  Corridor,  Range,  Bearing,  and  Radar.  The  value  of  the  ninth  attribute, 
IFF  (Identification:  Friend  or  Foe)  is  always  given  and  does  not  need  to  be  requested.  To  request  the 
value  of  an  attribute,  you  must  use  the  mouse  to  click  in  the  box  containing  the  name  of  that 
attribute. 

VAVR  Only 

The  CIC  will  verbally  acknowledge  your  request  for  an  attribute  value  through  the  headphones. 
The  ID  officer  will  acknowledge  your  request  by  saying  “ID  aye”  and  the  TRK  officer  will 
acknowledge  by  saying  “TRK  aye.” 

The  ID  and  TRK  officers  are  receiving  requests  from  other  Naval  personnel  and  command 
centers  at  the  same  time  you  are  querying  them.  If  they  have  heard  your  request  they  will 
immediately  acknowledge  it,  but  if  they  do  not  acknowledge  immediately,  then  you  should  assume 
they  have  not  heard  you  and  should  query  again.  The  CIC  must  sometimes  query  others  for  the 
answer  to  your  request,  so  if  they  have  acknowledged  your  request  just  wait  for  the  answer.  The  ID  or 
TRK  officer  will  verbally  deliver  the  attribute  value  through  the  headphones. 

If  it  is  helpful  to  you,  you  have  the  option  of  clicking  on  the  empty  text  box  to  the  right  of  the 
attribute  name  and  typing  in  the  attribute  value.  No  cursor  will  appear  in  this  box,  but  you  can  still 
type  there. 

You  must  verbally  acknowledge  receipt  of  the  attribute  information  by  saying  aloud  “the  name 
of  the  attribute  aye.”  For  example,  if  you  have  just  received  the  information  for  range,  you  would 
say,  “range  aye.”  If  you  fail  to  promptly  acknowledge  the  CIC’s  delivery  of  attribute  information, 
the  CIC  will  respond  with  an  alert.  It  will  be  displayed  in  a  window  in  the  lower  left  of  your  screen 
and  will  ask  if  you  have  received  the  information.  If  you  reply  YES  to  the  alert,  you  are  saying  that 
you  have  received  the  information.  If  you  reply  NO,  the  information  will  be  delivered  again.  You 
must  reply  to  this  alert  before  going  on  with  the  experiment. 

CAVR  Only 

Your  request  will  be  acknowledged  by  the  CIC  highlighting  the  Request  Ackned  box  briefly. 
The  Request  Ackned  box  on  the  ID  side  of  the  screen  will  become  highlighted  if  you  are  asking 
about  an  ID  attribute,  and  the  box  on  the  TRK  side  of  the  screen  will  become  highlighted  if  you  are 
asking  about  a  TRK  attribute. 
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The  ID  and  TRK  officers  are  receiving  requests  from  other  Naval  personnel  and  command 
centers  at  the  same  time  you  are  querying  them.  If  they  have  heard  your  request,  they  will 
immediately  acknowledge  it,  but  if  they  do  not  acknowledge  immediately,  then  you  should  assume 
they  have  not  heard  you  and  should  query  again.  The  CIC  must  sometimes  query  others  for  the 
answer  to  your  request,  so  if  they  have  acknowledged  your  request,  just  wait  for  the  answer.  The  ID  or 
TRK  officer  will  verbally  deliver  the  attribute  value  through  the  headphones. 

If  it  is  helpful  to  you,  you  have  the  option  of  clicking  on  the  empty  box  to  the  right  of  the 
attribute  name  and  typing  in  the  attribute  value.  No  cursor  will  appear  in  this  box,  but  you  can  still 
type  there. 

You  must  acknowledge  receipt  of  the  attribute  information  by  clicking  the  Send  Ackn  box 
corresponding  to  that  attribute.  If  you  fail  to  promptly  acknowledge  the  CIC’s  delivery  of  attribute 
information,  the  CIC  will  respond  with  an  alert.  It  will  be  displayed  in  a  window  in  the  lower  left  of 
your  screen  and  will  ask  if  you  have  received  the  information.  If  you  reply  YES  to  the  alert,  you  are 
saying  that  you  have  received  the  information.  If  you  reply  NO,  the  information  will  be  delivered 
again.  You  must  reply  to  this  alert  before  going  on  with  the  experiment. 

VACR  Only 

The  CIC  will  verbally  acknowledge  your  request  for  an  attribute  value  through  the  headphones. 
The  ID  officer  will  acknowledge  your  request  by  saying  “ID  aye,”  and  the  TRK  officer  will 
acknowledge  by  saying  “TRK  aye.” 

The  ID  and  TRK  officers  are  receiving  requests  from  other  Naval  personnel  and  command 
centers  at  the  same  time  you  are  querying  them.  If  they  have  heard  your  request  they  will 
immediately  acknowledge  it,  but  if  they  do  not  acknowledge  immediately,  then  you  should  assume 
they  have  not  heard  you  and  should  query  again.  The  CIC  must  sometimes  query  others  for  the 
answer  to  your  request,  so  if  they  have  acknowledged  your  request,  just  wait  for  the  answer.  The  ID  or 
TRK  officer’s  reply  will  appear  in  the  empty  box  to  the  right  of  the  attribute  name. 

You  must  verbally  acknowledge  receipt  of  the  attribute  information  by  saying  aloud  “the  name 
of  the  attribute  aye.”  For  example,  if  you  have  just  received  the  information  for  range,  you  would 
say,  “range  aye.”  If  you  fail  to  promptly  acknowledge  the  CIC’s  delivery  of  attribute  information, 
the  CIC  will  respond  with  an  alert.  It  will  be  displayed  in  a  window  in  the  lower  left  of  your  screen 
and  will  ask  if  you  have  received  the  information.  If  you  reply  YES  to  the  alert,  you  are  saying  that 
you  have  received  the  information.  If  you  reply  NO,  the  information  will  be  delivered  again.  You 
must  reply  to  this  alert  before  going  on  with  the  experiment. 

CACR  Only 

Your  request  will  be  acknowledged  by  the  CIC  highlighting  the  Request  Ackned  box  briefly. 
The  Request  Ackned  box  on  the  ID  side  of  the  screen  will  become  highlighted  if  you  are  asking 
about  an  ID  attribute  and  the  box  on  the  TRK  side  of  the  screen  will  become  highlighted  if  you  are 
asking  about  a  TRK  attribute. 

The  ID  and  TRK  officers  are  receiving  requests  from  other  Naval  personnel  and  command 
centers  at  the  same  time  you  are  querying  them.  If  they  have  heard  your  request,  they  will 
immediately  acknowledge  it,  but  if  they  do  not  acknowledge  immediately,  then  you  should  assume 
they  have  not  heard  you  and  should  query  again.  The  CIC  must  sometimes  query  others  for  the 
answer  to  your  request,  so  if  they  have  acknowledged  your  request,  just  wait  for  the  answer.  The  ID  or 
TRK  officer’s  reply  will  appear  in  the  empty  box  to  the  right  of  the  attribute  name. 

You  must  acknowledge  receipt  of  the  attribute  information  by  clicking  the  Send  Ackn  box 
corresponding  to  that  attribute.  If  you  fail  to  promptly  acknowledge  the  CIC’s  delivery  of  attribute 
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information,  the  CIC  will  respond  with  an  alert.  It  will  be  displayed  in  a  window  in  the  lower  left  of 
your  screen  and  will  ask  if  you  have  received  the  information.  If  you  reply  YES  to  the  alert,  you  are 
saying  that  you  have  received  the  information.  If  you  reply  NO,  the  information  will  be  delivered 
again.  You  must  reply  to  this  alert  before  going  on  with  the  experiment. 

All  Conditions 

You  must  determine  whether  each  attribute  value  is  Very  Threatening,  Somewhat  Threatening,  or 
Nonthreatening.  The  standard  classification  chart  in  the  lower  right  of  the  screen  will  help  you  to 
make  this  determination.  If  a  value  falls  between  threat  categories  shown  on  the  chart,  you  must 
decide  which  category  level  is  most  appropriate.  You  may  place  a  check  in  the  attribute  level  check 
box  “V,”  “S,”  or  “N”  to  indicate  whether  that  attribute  is  Very  Threatening,  Somewhat 
Threatening,  or  Nonthreatening,  respectively.  Checking  these  boxes  is  optional  and  is  provided  to 
help  you  remember  and  organize  your  attribute  threat  information. 

To  determine  how  you  will  interact  with  the  target,  you  must  use  the  equation  in  the  lower  left  of 
the  screen  to  calculate  a  numerical  value.  Attribute  threat  level  is  represented  by  a  cue  value  in  the 
equation  (e.g.,  “Altitude  Cue”).  An  attribute  that  is  Very  Threatening  is  assigned  a  numerical  value 
of  2;  Somewhat  Threatening  is  1;  and  Nonthreatening  is  0.  These  cue  values  are  shown  on  the  last 
line  of  the  chart  in  the  lower  right  of  the  screen. 

Please  focus  on  accuracy  in  this  task,  rather  than  speed. 
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Practice  1 

20000  ft 
-10  dgs 
700  mph 
13  m 

1.6  MHz 
13  mi 
25  mi 
28  dgs 

5  class 
21000  ft 
1  dgs 
200  mph 
60  m 
0.4  MHz 
13  mi 
75  mi 

6  dgs 
8  class 

Scenario  3 

30000  ft 

3  dgs 
650  mph 

60  m 
1.1  MHz 
28  mi 

61  mi 
24  dgs 
1  class 
18000  ft 
-3  dgs 
775  mph 
32  m 

1  MHz 
15  mi 
83  mi 

4  dgs 

5  class 


Scenario  1 

20000  ft 
-8  dgs 
300  mph 
12  m 
0.6  MHz 
12  mi 
125  mi 
28  dgs 

8  class 
7000  ft 
-3  dgs 
475  mph 
14  m 

1.6  MHz 
16  mi 
65  mi 

1  dg 

9  class 


Scenario  4 

6000  ft 
-11  dgs 
600  mph 
11  mi 

1  MHz 
22  mi 
75  mi 

2  dgs 

9  class 
12000  ft 

10  dgs 
710  mph 
32  m 
0.9  MHz 
26  mi 

15  mi 

7  dgs 

8  class 


Scenario  2 

11000  ft 
0  dgs 
625  mph 
15  m 

1.7  MHz 
24  mi 
22  mi 
5  dgs 

9  class 
21000  ft 
2  dgs 
650  mph 
27  m 

1  MHz 
18  mi 
80  mi 
13  dgs 
5  class 

Scenario  5 

23000  ft 
-15  dgs 
650  mph 

10  m 
1.1  MHz 
29  mi 
35  mi 
17  dgs 

2  class 
5000  ft 
1  dg 

250  mph 
26  m 
0.9  MHz 
5  mi 
64  mi 
12  dgs 
5  class 
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Scenario  6 

Scenario  7 

Scenario  8 

8000  ft 

29000  ft 

22000  ft 

-12  dgs 

-9  dgs 

-14  dgs 

625  mph 

740  mph 

775  mph 

13  m 

24  m 

30  m 

1.4  MHz 

1.5  MHz 

1.5  MHz 

30  mi 

23  mi 

25  mi 

30  mi 

175  mi 

14  mi 

8  dgs 

16  dgs 

6  dgs 

8  class 

9  class 

9  class 

19000  ft 

17000  ft 

23000  ft 

-8  dgs 

12  dgs 

-2  dgs 

700  mph 

400  mph 

475  mph 

35  m 

16  m 

52  m 

1.8  MHz 

0.4  MHz 

1.1  MHz 

13  mi 

15  mi 

15  mi 

5  mi 

72  mi 

87  mi 

3  dgs 

18  dgs 

14  dgs 

5  class 

2  class 

7  class 

Scenario  9 

Scenario  10 

9000  ft 

16000  ft 

-7  dgs 

-10  dgs 

790  mph 

750  mph 

11  m 

17  m 

1.1  MHz 

1.7  MHz 

27  mi 

25  mi 

9  mi 

32  mi 

4  dgs 

12  dgs 

8  class 

8  class 

13000  ft 

10000  ft 

1  dgs 

-2  dgs 

375  mph 

450  mph 

29  m 

14  m 

0.2  MHz 

0.9  MHz 

17  mi 

30  mi 

78  mi 

22  mi 

19  dgs 

15  dgs 

5  class 

4  class 

Appendix  C 

SAMPLE  DATA  FILES 


DATA  FILE  EXTRACTS 

Each  experiment  session  comprised  10  trials  for  which  four  data  files  (Subject  Snapshot,  Subject 
Summary,  Wizard  Snapshot,  and  Wizard  Summary)  were  generated.  The  following  excerpts  represent 
a  single  trial  for  one  participant. 

Subject  Snapshot  File  for  Participant  2113104 


Trial  =  1 

Time  =  188  Requests  wiz  1  threat  1  attr  1 

Time  =  283  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  1  attr  1 
Time  =  366  Subj  received  ackn  from  wiz  1  threat  1  attr  1 
Time  =  370  Requests  wiz  2  threat  1  attr  5 

Time  =  436  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  1  attr  5 
Time  =  552  Subj  received  ackn  from  wiz  2  threat  1  attr  5 
Time  =  717  Subj  received  reply  from  wiz  2  threat  1  attr  5 
Time  =  839  Subj  received  reply  from  wiz  1  threat  1  attr  1 
Time  =  842  Selected  priority  V  wiz  1  threat  1  attr  IFF 
Correct  threat  level  is  V 

Time  =  846  Entering  text  for  wiz  2  threat  1  attr  5:24 
Time  =  904  Entering  text  for  wiz  1  threat  1  attr  1:11000 
Time  =  1468  Selected  priority  S  wiz  1  threat  2  attr  IFF 
Correct  threat  level  is  S 
Time  =  1540  Requests  wiz  1  threat  2  attr  1 
Time  =  1600  Requests  wiz  2  threat  2  attr  5 

Time  =  1649  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  2  attr  1 

Time  =  1719  Subj  received  ackn  from  wiz  1  threat  2  attr  1 

Time  =  1 857  Subj  received  reply  from  wiz  1  threat  2  attr  1 

Time  =  1878  Entering  text  for  wiz  1  threat  2  attr  1:21000 

Time  =  2076  Requests  wiz  2  threat  2  attr  5 

Time  =  2139  Requests  wiz  1  threat  2  attr  2 

Time  =  2155  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  2  attr  5 

Time  =  2270  Subj  received  ackn  from  wiz  2  threat  2  attr  5 

Time  =  2424  Subj  received  reply  from  wiz  2  threat  2  attr  5 

Time  =  2424  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  2  attr  2 

Time  =  2493  Subj  received  ackn  from  wiz  1  threat  2  attr  2 

Time  =  2497  Entering  text  for  wiz  2  threat  2  attr  5:18 

Time  =  2784  Requests  wiz  2  threat  2  attr  6 

Time  =  2873  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  2  attr  6 
Time  =  2988  Subj  received  ackn  from  wiz  2  threat  2  attr  6 
Time  =  3093  Subj  received  reply  from  wiz  1  threat  2  attr  2 
Time  =  3121  Entering  text  for  wiz  1  threat  2  attr  2:2 
Time  =  3325  Subj  received  reply  from  wiz  2  threat  2  attr  6 
Time  =  3328  Entering  text  for  wiz  2  threat  2  attr  6:80 

Time  =  3767  Selected  priority  V  wiz  1  threat  1  attr  1 

Correct  threat  level  is  V 

Time  =  4071  Selected  priority  V  wiz  2  threat  1  attr  5 
Correct  threat  level  is  V 
Time  =  4146  Requests  wiz  1  threat  1  attr  2 
Time  =  421 1  Requests  wiz  2  threat  1  attr  6 

Time  =  4312  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  1  attr  2 
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Time  =  4381  Subj  received  ackn  from  wiz  1  threat  1  attr  2 

Time  =  4495  Subj  received  reply  from  wiz  1  threat  1  attr  2 

Time  =  4495  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  1  attr  6 
Time  =  4611  Subj  received  ackn  from  wiz  2  threat  1  attr  6 

Time  =  4767  Subj  received  reply  from  wiz  2  threat  1  attr  6 

Time  =  477 1  Selected  priority  S  wiz  1  threat  1  attr  2 
Correct  threat  level  is  S 

Time  =  5098  Selected  priority  V  wiz  2  threat  1  attr  6 
Correct  threat  level  is  V 
Time  =  5245  Requests  wiz  1  threat  1  attr  3 

Time  =  5291  Requests  wiz  2  threat  1  attr  7 

Time  =  5422  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  1  attr  3 
Time  =  5491  Subj  received  ackn  from  wiz  1  threat  1  attr  3 

Time  =  5647  Subj  received  reply  from  wiz  1  threat  1  attr  3 

Time  =  5647  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  1  attr  7 
Time  =  5763  Subj  received  ackn  from  wiz  2  threat  1  attr  7 

Time  =  5910  Subj  received  reply  from  wiz  2  threat  1  attr  7 

Time  =  5913  Entering  text  for  wiz  1  threat  1  attr  3:625 
Time  =  5992  Entering  text  for  wiz  2  threat  1  attr  7:5 
Time  =  6307  Selected  priority  V  wiz  1  threat  1  attr  3 
Correct  threat  level  is  V 

Time  =  6544  Selected  priority  V  wiz  2  threat  1  attr  7 
Correct  threat  level  is  V 
Time  =  6641  Requests  wiz  1  threat  1  attr  4 

Time  =  6711  Requests  wiz  2  threat  1  attr  8 

Time  =  6711  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  1  attr  4 
Time  =  6781  Subj  received  ackn  from  wiz  1  threat  1  attr  4 

Time  =  6781  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  1  attr  8 
Time  =  6897  Subj  received  ackn  from  wiz  2  threat  1  attr  8 

Time  =  7245  Subj  received  reply  from  wiz  2  threat  1  attr  8 

Time  =  7332  Subj  received  reply  from  wiz  1  threat  1  attr  4 

Time  =  7336  Entering  text  for  wiz  2  threat  1  attr  8:9 

Time  =  7376  Entering  text  for  wiz  1  threat  1  attr  4:15 
Time  =  7701  Selected  priority  V  wiz  1  threat  1  attr  4 
Correct  threat  level  is  V 

Time  =  7992  Selected  priority  V  wiz  2  threat  1  attr  8 
Correct  threat  level  is  V 
Time  =  9128  Decision  E  for  threat  1 
Correct  decision  is  E 

Time  =  945 1  Selected  priority  S  wiz  1  threat  2  attr  1 
Correct  threat  level  is  S 

Time  =  9737  Selected  priority  S  wiz  2  threat  2  attr  5 
Correct  threat  level  is  S 

Time  =  10056  Selected  priority  S  wiz  2  threat  2  attr  6 
Correct  threat  level  is  S 

Time  =  10311  Selected  priority  S  wiz  1  threat  2  attr  2 
Correct  threat  level  is  S 
Time  =  10368  Requests  wiz  1  threat  2  attr  3 

Time  =  10418  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  2  attr  3 

Time  =  10487  Subj  received  ackn  from  wiz  1  threat  2  attr  3 

Time  =  10635  Subj  received  reply  from  wiz  1  threat  2  attr  3 

Time  =  10640  Requests  wiz  2  threat  2  attr  7 

Time  =  10643  Entering  text  for  wiz  1  threat  2  attr  3:650 

Time  =  10724  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  2  attr  7 
Time  =  10839  Subj  received  ackn  from  wiz  2  threat  2  attr  7 
Time  =  11091  Selected  priority  V  wiz  1  threat  2  attr  3 
Correct  threat  level  is  V 
Time  =  11156  Requests  wiz  1  threat  2  attr  4 
Time  =  11376  Subj  received  reply  from  wiz  2  threat  2  attr  7 
Time  =  11379  Entering  text  for  wiz  2  threat  2  attr  7:13 
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Time  =  11380  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  2  attr  4 
Time  =  11449  Subj  received  ackn  from  wiz  1  threat  2  attr  4 
Time  =  11753  Subj  received  reply  from  wiz  1  threat  2  attr  4 
Time  =  11757  Selected  priority  S  wiz  2  threat  2  attr  7 
Correct  threat  level  is  S 
Time  =  1 1762  Requests  wiz  2  threat  2  attr  8 
Time  =  11820  Requests  wiz  1  threat  2  attr  4 
Time  =  11852  Entering  text  for  wiz  1  threat  2  attr  4:27 

Time  =  12019  Subj  Mach  knows  wiz  sending  first  response  for  wiz  1  threat  2  attr  4 
Time  =  12088  Subj  received  ackn  from  wiz  1  threat  2  attr  4 
Time  =  12234  Requests  wiz  2  threat  2  attr  8 

Time  =  12276  Subj  Mach  knows  wiz  sending  first  response  for  wiz  2  threat  2  attr  8 
Time  =  12391  Subj  received  ackn  from  wiz  2  threat  2  attr  8 
Time  =  12496  Subj  received  reply  from  wiz  1  threat  2  attr  4 
Time  =  12755  Selected  priority  S  wiz  1  threat  2  attr  4 
Correct  threat  level  is  S 

Time  =  12950  Subj  received  reply  from  wiz  2  threat  2  attr  8 
Time  =  13430  Selected  priority  S  wiz  2  threat  2  attr  8 
Correct  threat  level  is  S 
Time  =  14611  Decision  W  for  threat  2 
Correct  decision  is  W 

Time  =  14621  Subj  Mach  told  to  clean  up  trial. 

Time  =  14648  Successfully  ended  trial. 

Subject  Summary  File  for  Participant  2113104 

Trial  =  1 

Mode  =  VAVR 

Target  1  decision  is  Engage 

Target  2  decision  is  Watch 

Attr  1-9  are  threat  1  and  attr  10-18  are  threat  2. 

Attributes  15,  8,  0  and  12  have  a  short  delay. 

Attributes  10,  16,  17  and  3  have  a  long  delay. 

Elapsed  time=  14611 
The  decision  for  threat  1  was  E 
The  decision  for  threat  2  was  W 
Requests  for  threat  1  attr  1=1 
Requests  for  threat  1  attr  2=1 
Requests  for  threat  1  attr  3=1 
Requests  for  threat  1  attr  4=1 
Requests  for  threat  1  attr  5=1 
Requests  for  threat  1  attr  6=1 
Requests  for  threat  1  attr  7=1 
Requests  for  threat  1  attr  8=1 
Requests  for  threat  2  attr  1=1 
Requests  for  threat  2  attr  2=1 
Requests  for  threat  2  attr  3=1 
Requests  for  threat  2  attr  4=2 
Requests  for  threat  2  attr  5=2 
Requests  for  threat  2  attr  6=1 
Requests  for  threat  2  attr  7=1 
Requests  for  threat  2  attr  8=2 
Priority  selected  for  threat  1  attr  1=1 
Priority  selected  for  threat  1  attr  2=1 
Priority  selected  for  threat  1  attr  3=1 
Priority  selected  for  threat  1  attr  4=1 
Priority  selected  for  threat  1  attr  5=1 
Priority  selected  for  threat  1  attr  6=1 
Priority  selected  for  threat  1  attr  7=1 
Priority  selected  for  threat  1  attr  8=1 
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Priority  selected  for  threat  2  attr  1=1 
Priority  selected  for  threat  2  attr  2=1 
Priority  selected  for  threat  2  attr  3=1 
Priority  selected  for  threat  2  attr  4=1 
Priority  selected  for  threat  2  attr  5=1 
Priority  selected  for  threat  2  attr  6=1 
Priority  selected  for  threat  2  attr  7=1 
Priority  selected  for  threat  2  attr  8=1 
Priority  changes  for  threat  1  attr  1  =0 
Priority  changes  for  threat  1  attr  2=0 
Priority  changes  for  threat  1  attr  3=0 
Priority  changes  for  threat  1  attr  4=0 
Priority  changes  for  threat  1  attr  5=0 
Priority  changes  for  threat  1  attr  6=0 
Priority  changes  for  threat  1  attr  7=0 
Priority  changes  for  threat  1  attr  8=0 
Priority  changes  for  threat  2  attr  1=0 
Priority  changes  for  threat  2  attr  2=0 
Priority  changes  for  threat  2  attr  3=0 
Priority  changes  for  threat  2  attr  4=0 
Priority  changes  for  threat  2  attr  5=0 
Priority  changes  for  threat  2  attr  6=0 
Priority  changes  for  threat  2  attr  7=0 
Priority  changes  for  threat  2  attr  8=0 
Accessed  text  box  for  threat  1  attr  1=1 
Accessed  text  box  for  threat  1  attr  2=0 
Accessed  text  box  for  threat  1  attr  3=1 
Accessed  text  box  for  threat  1  attr  4=1 
Accessed  text  box  for  threat  1  attr  5=1 
Accessed  text  box  for  threat  1  attr  6=0 
Accessed  text  box  for  threat  1  attr  7=1 
Accessed  text  box  for  threat  1  attr  8=1 
Accessed  text  box  for  threat  2  attr  1=1 
Accessed  text  box  for  threat  2  attr  2=1 
Accessed  text  box  for  threat  2  attr  3=1 
Accessed  text  box  for  threat  2  attr  4=1 
Accessed  text  box  for  threat  2  attr  5=1 
Accessed  text  box  for  threat  2  attr  6=1 
Accessed  text  box  for  threat  2  attr  7=1 
Accessed  text  box  for  threat  2  attr  8=0 
Wizard  Snapshot  File  for  Participant  2113104 

Trial  =  1 
Mode=  VAVR 

Time  =  187  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  1  attr  1 

Time  =  278  Reply  Button  wiz  1  threat  1  attr  1 

Time  =  280  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  1  attr  1 

Time  =  370  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  1  attr  5 

Time  =  434  Reply  Button  wiz  2  threat  1  attr  5 

Time  =  435  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  1  attr  5 

Time  =  858  Ackn  Button  wiz  2  threat  1  attr  5 

Time  =  964  Ackn  Button  wiz  1  threat  1  attr  1 

Time  =1218  Mach  knows  to  set  alert  for  wiz  2  threat  1  attr  5 

Time  =1218  Mach  knows  to  cancel  alert  for  wiz  2  threat  1  attr  5 

Time  =  1339  Mach  knows  to  set  alert  for  wiz  1  threat  1  attr  1 

Time  =  1340  Mach  knows  to  cancel  alert  for  wiz  1  threat  1  attr  1 

Time  =1541  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  2  attr  1 

Time  =1601  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  2  attr  5  but  denied  because  2nd 

request  attr. 

Time  =  1646  Reply  Button  wiz  1  threat  2  attr  1 
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Time  =  1648  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  2  attr  1 
Time  =  1976  Ackn  Button  wiz  1  threat  2  attr  1 


Time  =  2077  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  2  attr  5 

Time  =  2140  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  2  attr  2 

Time  =  2153  Reply  Button  wiz  2  threat  2  attr  5 

Time  =  2155  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  2  attr  5 

Time  =  2248  Reply  Button  wiz  1  threat  2  attr  2 

Time  =  2250  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  2  attr  2 

Time  =  2496  Mach  knows  to  set  alert  for  wiz  1  threat  2  attr  1 

Time  =  2497  Mach  knows  to  cancel  alert  for  wiz  1  threat  2  attr  1 

Time  =  2557  Ackn  Button  wiz  2  threat  2  attr  5 

Time  =  2787  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  2  attr  6 

Time  =  2874  Reply  Button  wiz  2  threat  2  attr  6 

Time  =  2875  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  2  attr  6 

Time  =  3098  Mach  knows  to  set  alert  for  wiz  2  threat  2  attr  5 

Time  =  3099  Mach  knows  to  cancel  alert  for  wiz  2  threat  2  attr  5 

Time  =  3182  Ackn  Button  wiz  1  threat  2  attr  2 

Time  =  3391  Ackn  Button  wiz  2  threat  2  attr  6 

Time  =  3599  Mach  knows  to  set  alert  for  wiz  1  threat  2  attr  2 

Time  =  3599  Mach  knows  to  cancel  alert  for  wiz  1  threat  2  attr  2 

Time  =  3829  Mach  knows  to  set  alert  for  wiz  2  threat  2  attr  6 

Time  =  3830  Mach  knows  to  cancel  alert  for  wiz  2  threat  2  attr  6 

Time  =  4151  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  1  attr  2 

Time  =  4215  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  1  attr  6 

Time  =  4314  Reply  Button  wiz  1  threat  1  attr  2 

Time  =  4316  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  1  attr  2 

Time  =  4369  Reply  Button  wiz  2  threat  1  attr  6 

Time  =  4371  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  1  attr  6 

Time  =  4625  Ackn  Button  wiz  1  threat  1  attr  2 

Time  =  4943  Ackn  Button  wiz  2  threat  1  attr  6 

Time  =  5000  Mach  knows  to  set  alert  for  wiz  1  threat  1  attr  2 

Time  =  5001  Mach  knows  to  cancel  alert  for  wiz  1  threat  1  attr  2 

Time  =  525 1  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  1  attr  3 

Time  =  5274  Mach  knows  to  set  alert  for  wiz  2  threat  1  attr  6 

Time  =  5274  Mach  knows  to  cancel  alert  for  wiz  2  threat  1  attr  6 

Time  =  5297  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  1  attr  7 

Time  =  5425  Reply  Button  wiz  1  threat  1  attr  3 

Time  =  5426  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  1  attr  3 

Time  =  5503  Reply  Button  wiz  2  threat  1  attr  7 

Time  =  5505  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  1  attr  7 

Time  =  5866  Ackn  Button  wiz  1  threat  1  attr  3 

Time  =  6016  Ackn  Button  wiz  2  threat  1  attr  7 

Time  =  6155  Mach  knows  to  set  alert  for  wiz  1  threat  1  attr  3 

Time  =  6155  Mach  knows  to  cancel  alert  for  wiz  1  threat  1  attr  3 

Time  =  6418  Mach  knows  to  set  alert  for  wiz  2  threat  1  attr  7 

Time  =  6418  Mach  knows  to  cancel  alert  for  wiz  2  threat  1  attr  7 

Time  =  6649  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  1  attr  4 

Time  =  6713  Reply  Button  wiz  1  threat  1  attr  4 

Time  =  6715  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  1  attr  4 

Time  =  6720  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  1  attr  8 

Time  =  6777  Reply  Button  wiz  2  threat  1  attr  8 

Time  =  6779  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  1  attr  8 

Time  =  7340  Ackn  Button  wiz  2  threat  1  attr  8 

Time  =  7429  Ackn  Button  wiz  1  threat  1  attr  4 

Time  =  7754  Mach  knows  to  set  alert  for  wiz  2  threat  1  attr  8 

Time  =  7755  Mach  knows  to  cancel  alert  for  wiz  2  threat  1  attr  8 

Time  =  7841  Mach  knows  to  set  alert  for  wiz  1  threat  1  attr  4 

Time  =  7842  Mach  knows  to  cancel  alert  for  wiz  1  threat  1  attr  4 

Time  =  10377  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  2  attr  3 

Time  =  10422  Reply  Button  wiz  1  threat  2  attr  3 
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Time  =  10424  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  2  attr  3 

Time  =  10648  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  2  attr  7 

Time  =  10730  Reply  Button  wiz  2  threat  2  attr  7 

Time  =  10731  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  2  attr  7 

Time  =  10788  Ackn  Button  wiz  1  threat  2  attr  3 

Time  =  11146  Mach  knows  to  set  alert  for  wiz  1  threat  2  attr  3 

Time  =  11146  Mach  knows  to  cancel  alert  for  wiz  1  threat  2  attr  3 

Time  =  11167  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  2  attr  4 

Time  =  1 1246  Reply  Button  wiz  1  threat  2  attr  4 

Time  =  11248  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  2  attr  4 

Time  =  11471  Ackn  Button  wiz  2  threat  2  attr  7 

Time  =  11772  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  2  attr  8  but  denied  because  2nd 
request  attr. 

Time  =  11 830  Mach  knows  subj  asks  for  attr  value  for  wiz  1  threat  2  attr  4 

Time  =  11886  Mach  knows  to  set  alert  for  wiz  2  threat  2  attr  7 

Time  =  11887  Mach  knows  to  cancel  alert  for  wiz  2  threat  2  attr  7 

Time  =  11977  Ackn  Button  wiz  1  threat  2  attr  4 

Time  =  12024  Reply  Button  wiz  1  threat  2  attr  4 

Time  =  12028  Mach  sends  Reply  &  ackn  to  wiz  1  for  threat  2  attr  4 

Time  =  12245  Mach  knows  subj  asks  for  attr  value  for  wiz  2  threat  2  attr  8 

Time  =  12263  Mach  knows  to  set  alert  for  wiz  1  threat  2  attr  4 

Time  =  12264  Mach  knows  to  cancel  alert  for  wiz  1  threat  2  attr  4 

Time  =  12283  Reply  Button  wiz  2  threat  2  attr  8 

Time  =  12285  Mach  sends  Reply  &  ackn  to  wiz  2  for  threat  2  attr  8 

Time  =  1 3007  Mach  knows  to  set  alert  for  wiz  1  threat  2  attr  4 

Time  =  13008  Mach  knows  to  cancel  alert  for  wiz  1  threat  2  attr  4 

Time  =  13081  Ackn  Button  wiz  2  threat  2  attr  8 

Time  =  1 3204  Ackn  Button  wiz  1  threat  2  attr  4 

Time  =  13461  Mach  knows  to  set  alert  for  wiz  2  threat  2  attr  8 

Time  =  13462  Mach  knows  to  cancel  alert  for  wiz  2  threat  2  attr  8 

Time  =  14632  Mach  knows  subj  made  both  decisions. 

Time  =  15277  Ending  Trial  1 


Wizard  Summary  File  for  Participant  2113104 


Trial  =  1 
Mode=  VAVR 

Threat  2  Attr  8  and  Threat  2  Attr  5  for  Wiz  2 
require  a  second  request  before  reply  is  made. 
Number  of  alarms  for  Wiz  1=  0 
Number  of  alarms  for  Wiz  2=  0 
Number  of  AttrRequest  for  threat  1  and  attr  1  =  1 

Number  of  AttrRequest  for  threat  1  and  attr  2  =  1 

Number  of  AttrRequest  for  threat  1  and  attr  3  =  1 

Number  of  AttrRequest  for  threat  1  and  attr  4=1 
Number  of  AttrRequest  for  threat  1  and  attr  5  =  1 

Number  of  AttrRequest  for  threat  1  and  attr  6  =  1 

Number  of  AttrRequest  for  threat  1  and  attr  7  =  1 

Number  of  AttrRequest  for  threat  1  and  attr  8  =  1 

Number  of  AttrRequest  for  threat  2  and  attr  1  =  1 

Number  of  AttrRequest  for  threat  2  and  attr  2  =  1 

Number  of  AttrRequest  for  threat  2  and  attr  3  =  1 

Number  of  AttrRequest  for  threat  2  and  attr  4  =  2 

Number  of  AttrRequest  for  threat  2  and  attr  5  =  2 

Number  of  AttrRequest  for  threat  2  and  attr  6  =  1 

Number  of  AttrRequest  for  threat  2  and  attr  7  =  1 

Number  of  AttrRequest  for  threat  2  and  attr  8  =  2 

Number  of  AcknRec  selected  for  threat  1  attr  1  =  1 

Number  of  AcknRec  selected  for  threat  1  attr  2  =  1 


Number  of  AcknRec  selected  for  threat  1  attr  3  =  1 
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Number  of  AcknRec  selected  for  threat  1  attr  4  =  1 

Number  of  AcknRec  selected  for  threat  1  attr  5  =  1 

Number  of  AcknRec  selected  for  threat  1  attr  6=1 

Number  of  AcknRec  selected  for  threat  1  attr  7  =  1 

Number  of  AcknRec  selected  for  threat  1  attr  8  =  1 

Number  of  AcknRec  selected  for  threat  2  attr  1  =  1 

Number  of  AcknRec  selected  for  threat  2  attr  2=1 

Number  of  AcknRec  selected  for  threat  2  attr  3  =  1 

Number  of  AcknRec  selected  for  threat  2  attr  4  =  2 

Number  of  AcknRec  selected  for  threat  2  attr  5  =  1 

Number  of  AcknRec  selected  for  threat  2  attr  6  =  1 

Number  of  AcknRec  selected  for  threat  2  attr  7  =  1 

Number  of  AcknRec  selected  for  threat  2  attr  8  =  1 

The  next  group  will  only  have  values  if  mode  is  CA  or  if  alert  in  VA. 

Number  of  AcknRec  highlited  for  threat  1  attr  1  =  1 

Number  of  AcknRec  highlited  for  threat  1  attr  2=1 

Number  of  AcknRec  highlited  for  threat  1  attr  3  =  1 

Number  of  AcknRec  highlited  for  threat  1  attr  4=1 

Number  of  AcknRec  highlited  for  threat  1  attr  5  =  1 

Number  of  AcknRec  highlited  for  threat  1  attr  6  =  1 

Number  of  AcknRec  highlited  for  threat  1  attr  7  =  1 

Number  of  AcknRec  highlited  for  threat  1  attr  8  =  1 

Number  of  AcknRec  highlited  for  threat  2  attr  1  =  1 

Number  of  AcknRec  highlited  for  threat  2  attr  2=1 

Number  of  AcknRec  highlited  for  threat  2  attr  3  =  1 

Number  of  AcknRec  highlited  for  threat  2  attr  4  =  2 

Number  of  AcknRec  highlited  for  threat  2  attr  5  =  1 

Number  of  AcknRec  highlited  for  threat  2  attr  6=1 

Number  of  AcknRec  highlited  for  threat  2  attr  7  =  1 

Number  of  AcknRec  highlited  for  threat  2  attr  8  =  1 

Number  of  Alerts  highlited  threat  1  attr  1=0 

Number  of  Alerts  highlited  threat  1  attr  2  =  0 

Number  of  Alerts  highlited  threat  1  attr  3  =  0 

Number  of  Alerts  highlited  threat  1  attr  4  =  0 

Number  of  Alerts  highlited  threat  1  attr  5  =  0 

Number  of  Alerts  highlited  threat  1  attr  6  =  0 

Number  of  Alerts  highlited  threat  1  attr  7  =  0 

Number  of  Alerts  highlited  threat  1  attr  8  =  0 

Number  of  Alerts  highlited  threat  2  attr  1  =  0 

Number  of  Alerts  highlited  threat  2  attr  2  =  0 

Number  of  Alerts  highlited  threat  2  attr  3  =  0 

Number  of  Alerts  highlited  threat  2  attr  4  =  0 

Number  of  Alerts  highlited  threat  2  attr  5  =  0 

Number  of  Alerts  highlited  threat  2  attr  6  =  0 

Number  of  Alerts  highlited  threat  2  attr  7  =  0 

Number  of  Alerts  highlited  threat  2  attr  8  =  0 

Number  of  WizReplies  for  threat  1  attr  1  =  1 

Number  of  WizReplies  for  threat  1  attr  2=1 

Number  of  WizReplies  for  threat  1  attr  3=1 

Number  of  WizReplies  for  threat  1  attr  4=1 

Number  of  WizReplies  for  threat  1  attr  5  =  1 

Number  of  WizReplies  for  threat  1  attr  6=1 

Number  of  WizReplies  for  threat  1  attr  7  =  1 

Number  of  WizReplies  for  threat  1  attr  8  =  1 

Number  of  WizReplies  for  threat  2  attr  1  =  1 

Number  of  WizReplies  for  threat  2  attr  2=1 

Number  of  WizReplies  for  threat  2  attr  3=1 

Number  of  WizReplies  for  threat  2  attr  4  =  2 

Number  of  WizReplies  for  threat  2  attr  5  =  1 

Number  of  WizReplies  for  threat  2  attr  6=1 
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Number  of  WizReplies  for  threat  2  attr  7  =  1 
Number  of  WizReplies  for  threat  2  attr  8  =  1 
The  next  group  will  only  have  values  if  mode  is  VA. 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  1  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  2  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  3  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  4  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  5  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  6  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  7  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  1  attr  8  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  1  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  2  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  3  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  4  =  2 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  5  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  6=1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  7  =  1 

Number  of  Subj  verbal  Ackns  to  Wiz  for  threat  2  attr  8  =  1 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  1  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  2  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  3  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  4  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  5  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  6  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  7  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  1  attr  8  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  1=0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  2  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  3  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  4  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  5=0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  6  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  7  =  0 

Number  of  things  the  subject  heard  but  didn’t  ackn  for  threat  2  attr  8  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  1  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  2  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  3  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  4  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  5  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  6  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  7  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  1  attr  8  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  1  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  2  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  3  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  4  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  5  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  6  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  7  =  0 

Number  of  things  the  subject  did  not  hear  and  didn’t  ackn  for  threat  2  attr  8  =  0 


Appendix  D 

PROGRAM  EXECUTION 


NETWORKING  INSTRUCTIONS 

The  experiment  application  was  compiled  under  the  Macintosh  System  7  operating  system.  The 
following  descriptions  assume  operation  in  that  environment. 

To  run  in  experiment  mode,  open  the  Control  Panel  Network  and  select  LocalTalk  Built-in.  Then 
click  OK  in  the  dialog  box  concerning  network  connections.  Open  the  Control  Panel  MacTCP  and 
select  LocalTalk.  Finally  restart  the  machine. 

To  switch  to  experiment  mode  from  Open  Transport,  run  the  Network  Software  Selector  utility 
and  select  Classic  Networking.  Restart  the  machine  and  perform  the  same  operations  indicated  above 
to  run  in  experiment  mode. 

To  run  in  internet  mode  (network  with  no  experiment),  open  the  Control  Panel  Network  and 
select  EtherTalk.  Then  click  OK  in  the  dialog  box  concerning  network  connections.  Open  the  Control 
Panel  MacTCP  and  select  Ethernet.  Note  that  is  EtherNET,  not  EtherTALK.  Finally  restart  the 
machine. 

Given  the  size  and  amount  of  information  displayed  on  the  screen,  the  subject  code  should  be 
run  on  a  Macintosh  with  at  least  a  20-inch  screen  and  the  wizard  code  should  be  ran  on  a  Macintosh 
with  at  least  a  17-inch  screen. 

First  Time  Execution 

When  running  the  software  for  the  first  time  on  a  set  of  machines,  delete  the  Wizard.pref  and 
Subject.pref  files  from  the  Wizard  and  Subject  folders  on  their  respective  machines. 

On  the  Subject  machine: 

1 .  Double-click  the  Subject  icon.  A  dialog  box  will  appear  requesting  the  Subject  ID  number. 
Enter  an  ID  of  eight  or  fewer  characters. 

2.  The  Connection  Settings  dialog  box  appears.  Select  the  AppleTalk  ADSP  Tool  as  the 
method. 

3 .  In  the  subsequent  dialog  box,  the  Connection  Type  will  display  ADSP.  Enter  your  name  in 
the  Name  field  and  click  OK. 

4.  Select  Practice  or  Real  Trial  from  the  Type  Trial  menu. 

5 .  Select  Listen  from  the  Connection  menu. 

On  the  Wizard  machine: 

6 .  Double-click  on  the  Wizard  icon.  A  dialog  box  will  appear  requesting  the  Subject  ID.  Enter 
the  same  ID  used  on  the  Subject  machine. 

7.  The  Connection  Settings  dialog  box  appears.  Select  the  AppleTalk  ADSP  Tool  as  the 
method. 

8.  In  the  subsequent  dialog  box,  the  name  entered  on  the  Subject’s  Connections  Settings  will 
be  visible  in  the  box  under  the  Name  field.  Click  on  that  name  and  it  will  appear  in  the 
Name  field.  Click  OK. 
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9.  Select  VAVR ,  CAVR ,  VACR ,  or  CACR  from  the  Mode  menu.  This  will  cause  an  empty 
window  to  appear,  titled  to  match  the  mode  selected. 

10.  Select  Connect  from  the  Connection  menu. 

1 1 .  Select  Start  Trial  from  the  File  menu. 

The  Wizard  machine  will  now  display  a  window  with  four  buttons  (Reply,  Repeat,  AcknRec,  and 
Alert)  for  each  of  the  eight  attributes  of  the  two  targets.  The  Subject  machine  should  display  a 
window  with  various  buttons,  check  boxes,  and  text  boxes  for  each  of  the  attributes  as  well  as  radio 
buttons  for  decision  and  a  crib  sheet  and  formula  for  making  the  decisions.  Each  trial  is  completed 
when  the  participant  makes  both  decisions.  At  the  end  of  the  practice  sessions  and  the  experiment 
sessions,  a  dialog  box  appears  indicating  the  decisions  made  by  the  participant  and  the  correct 
decisions. 

Once  a  complete  session  has  executed,  the  Wizard.pref  and  Subject.pref  files  will  appear  in  the 
Wizard  and  Subject  folder.  If  they  appear  on  the  desktop  instead  and  a  connection  cannot  be  made 
between  the  machines  (a  bell  will  sound),  then  recompile  the  code.  Once  the  prefs  files  are  in  the 
correct  folder,  execution  of  the  Wizard  and  the  Subject  will  bypass  the  Connection  Settings  dialog 
boxes.  If  at  any  time  a  session  needs  to  be  terminated  early,  select  Quit  from  the  File  menu  of  either 
machine.  Then  select  Disconnect  from  the  Connection  menu  and  Quit  from  the  File  menu  of  the 
other  machine.  Failure  to  Disconnect  the  second  machine  will  corrupt  the  preference  files;  they  will 
need  to  be  deleted  and  new  ones  created. 


